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WORK-CENTERED  DESIGN  AND  EVALUATION 
OF  A  C2  VISUALIZATION  AID 


Emilie  Rothf,  Mona  Stilson§,  Ronald  Scotty,  Randall  Whitaker-, 
Tom  Kazmierczakt,  Gina  Thomas-Meyers§,  and  Jeffrey  Wampler§ 


t  Roth  Cognitive  Engineering,  Brookline,  MA 
$  BBN  Technologies,  Cambridge,  MA 
-  Northrop  Grumman  Information  Technology,  OH 
§Air  Force  Research  Laboratory,  Wright-Patterson  AFB,  OH 

Command  and  Control  (C2)  operators  increasingly  need  to  assimilate  large  amounts  of 
near-real  time  data  distributed  across  multiple  sources  to  identify,  interpret,  and  mentally 
fuse  the  information  necessary  to  accomplish  their  work.  We  have  been  developing  and 
applying  work-centered  design  and  evaluation  methodologies  to  design  advanced 
visualization  and  support  tools  intended  to  more  effectively  support  C2  cognitive  and 
collaborative  work.  The  paper  reports  the  results  of  a  work-centered  evaluation 
assessing  the  usability  and  usefulness  of  an  innovative  work-centered  visualization  aid  (a 
graphic  mission  timeline  display)  we  developed  to  support  mission  replanning  during 
execution  in  a  C2  airlift  service.  The  evaluation  compared  performance  with  the  work- 
centered  visualization  to  performance  using  the  existing  information  technology  system. 
The  work-centered  visualization  produced  statistically  significant  improvement  in  task 
completion  time,  errors,  workload  and  situation  awareness.  The  results  point  to  the  value 
of  taking  a  work-centered  analysis  and  design  approach. 


INTRODUCTION 

Command  and  Control  (C2)  operators  are 
increasingly  faced  with  the  need  to  assimilate  large  amounts  of 
near-real  time  data.  Too  often  the  needed  information  is 
distributed  across  a  variety  of  sources  requiring  operators  to 
find,  interpret  and  mentally  fuse  disparate  data  in  order  to 
successfully  accomplish  their  work.  A  challenge  for  the 
Cognitive  Systems  Engineering  community  is  to  develop  and 
disseminate  more  effective  methods  for  addressing  cognitive 
and  work-centered  issues  during  C2  system  design, 
development  and  evaluation. 

Over  the  last  several  years,  we  have  been  developing 
and  applying  work-centered  design  and  evaluation 
methodologies  to  the  design  of  advanced  visualization  and 
support  tools  intended  to  support  cognitive  and  collaborative 
work  within  a  C2  airlift  service  organization  (e.g.,  Eggleston 
and  Whitaker,  2002;  Eggleston,  2003;  Eggleston,  et  al.,  2005; 
Scott,  et  al.,  2005;  Wampler,  et  al.,  2005). 

Key  elements  of  work-centered  design  include:  (a) 

An  analysis  and  modeling  of  the  demands  of  work,  (b)  the 
design  of  displays/visualizations  that  reveal 
constraints/affordances  coupled  with  ‘direct  aiding’  support 
that  utilizes  ‘machine  intelligence’  to  integrate  data  into 
meaningful  information  in  the  context  of  the  work,  and  (c)  the 
use  of  work-centered  evaluations  -  evaluations  that  attempt  to 
probe  the  ability  of  the  support  system  to  support  the  work 
across  a  representative  range  of  work  context  and  complexities 
(Eggleston,  Roth  &  Scott,  2003;  Roth,  et  al.,  2002).  The  focus 
is  on  evaluating  not  only  the  ‘usability’  of  the  system  (the  ease 
with  which  one  can  learn  and  use  it)  but  also  the  ‘usefulness’ 
(the  extent  to  which  it  facilitates  performance  of  work)  and 


‘impact’  (the  extent  to  which  it  supports  the  work  goals  of  the 
individual,  the  immediate  work  group,  and  the  organization). 

In  this  paper  we  present  the  results  of  a  recent  project 
illustrating  our  work-centered  approach.  The  paper  reports  the 
results  of  a  work-centered  evaluation  conducted  to  assess  the 
usability  and  usefulness  of  a  prototype  visualization  aid 
designed  to  support  mission  replanning  in  a  military  airlift 
organization.  Wampler  et  al.  (2005)  provides  a  more  detailed 
account  of  the  cognitive  analyses  and  design  activities  that 
were  conducted  in  support  of  the  design  of  this  visualization 
aid. 

Overview  of  the  Context  of  Work 

The  military  airlift  organization  is  an  air  operations 
center  (AOC)  responsible  for  the  scheduling  and  tracking  of 
airlift  and  air  refueling  missions  worldwide.  Mission  planning 
is  a  complicated  activity  that  must  take  into  account  issues 
such  as  matching  loads  to  currently  available  aircraft,  landings 
in  and  over-flights  of  foreign  nations,  competing  airlift 
demands,  airfield  constraints,  air  refueling  requirements,  and 
aircrew  constraints. 

Twenty-four  hours  prior  to  a  planned  mission  launch, 
responsibility  for  the  mission  is  transferred  from  mission 
planners  to  the  Execution  Cell  that  is  responsible  for  handling 
last  minute  changes  and  problems  that  might  arise  during 
mission  execution.  The  existing  information  technology  (IT) 
systems  available  in  the  Execution  Cell  do  not  effectively 
support  them  in  easily  understanding  the  flight  plan,  assessing 
the  impact  of  changes  on  the  viability  of  the  flight  plan  and 
revising  the  plan  appropriately.  Although  all  the  relevant  data 
is  available,  it  is  presented  in  a  tabular  form  that  requires 
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navigation  across  multiple  tabular  displays  to  extract  and 
mentally  collate  the  necessary  information.  (See  Figure  1). 


Figure  I.  Example  of  the  existing  IT  system  tabular  displays. 

An  analysis  of  the  work  domain  was  conducted  based 
on  interviews  and  observations  of  mission  planners  and 
Execution  Cell  personnel  in  their  work  environment.  We 
documented  the  ‘as  is’  mission  planning  and  execution 
process,  the  factors  that  complicate  planning  and  execution, 
and  the  kinds  of  miscommunications  and  errors  that  can 
produce  mission  delays  and  cancellations.  A  number  of 
‘leverage  points’,  or  opportunities  for  more  effective  work- 
centered  support,  were  identified,  including: 

•  More  effectively  communicating  mission  plan  objectives, 
details,  and  constraints  to  the  personnel  overseeing 
mission  execution; 

•  Alerting  users  to  emerging  problems  (e.g.  delays)  that 
threaten  the  viability  of  a  mission; 

•  Facilitating  the  ability  to  assess  repercussions  of  mission 
changes  (e.g.,  delays)  on  the  current  and  subsequent 
missions  (e.g.,  reaching  an  airfield  after  hours;  violating 
crew  rest  requirements;  violating  diplomatic  clearance 
time  limits). 

These  three  elements  of  work-centered  support  correspond  to 
support  for  different  levels  of  situation  awareness  (SA)  with 
respect  to  mission  plans.  This  includes  support  for  the  three 
levels  of  SA  as  defined  by  Endsley  (1995);  understanding  the 
elements  of  a  mission  plan  (Level  1  SA);  understanding  the 
current  situation  with  respect  to  delays  and  impact  on  the 
current  sortie  (Level  2  SA);  and  projecting  impact  on  future 
sorties  and  future  missions  (Level  3  SA). 

Timeline  Prototype 

Our  proposal  for  a  work-centered  innovation  was  a 
timeline  tool  that  enables  users  to  visualize  the  temporal 
characteristics  and  constraints  of  a  mission  plan.  The 


objective  was  to  provide  visibility  into  decision  factors  that 
impact  mission  viability  so  that  users  can  readily  assess  the 
impact  of  changes  (e.g.,  delays)  on  the  viability  of  the  mission. 
A  second,  related  objective,  was  to  support  replan  decisions 
by  providing  a  ‘what  if  capability  that  allows  users  to  make 
changes  to  the  mission  plan  and  directly  see  the  impact. 

A  timeline  prototype  was  developed  that  enables 
Execution  Cell  personnel  to  “see  at  a  glance”  the  relationships 
between  mission  plan  elements  and  resource  constraints. 

Alerts  are  integrated  into  the  visualization  to  highlight 
exceptions  in  work  context  and  guide  problem-solving.  Highly 
critical  factors  (i.e.,  scheduled  departure,  legs  of  amission, 
origin  and  destination,  ports,  air  refueling,  etc.)  are  displayed 
in  a  “core”  visualization  and  visually  correlated  on  a 
timescale.  This  core  is  available  in  both  a  multi-mission  view 
(see  Figure  2)  and  an  individual  detailed  mission  view  (See 
Figure  3).  The  multi-mission  view  affords  summary  SA  on 
the  overall  workstream  and  alerts  the  operator  to  problems  that 
need  further  investigation.  The  detailed  mission  view  contains 
additional  relevant  factors  organized  into  “clusters”  (i.e., 
airspace,  aircrew,  airfield,  etc).  The  core  and  clusters  were 
derived  from  the  analysis  of  the  intrinsic  work  structure  and 
mental  models  utilized  in  mission  execution  work. 


Figure  3.  The  timeline  prototype  detailed  mission  view 

An  active  “what-if  ’  mode  is  available  in  the  detailed 
view  to  help  assess  alternative  courses  of  action  and  the  future 
repercussions  associated  with  each  alternative.  The  ‘what-if’ 
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mode  allows  the  user  to  make  changes  to  the  mission  plan  via 
direct  manipulation  features.  A  ‘validation’  function  is 
provided  that  checks  for  resulting  constraint  violations.  Any 
constraints  that  are  violated  will  trigger  an  alert.  The  user  can 
then  make  further  simulated  changes  until  a  workable  course 
of  action  is  found  that  will  lead  to  a  viable  plan. 

Work-Centered  Evaluation 

As  part  of  the  work-centered  development  process, 
we  conducted  a  work-centered  evaluation  of  the  timeline 
prototype.  As  Woods  and  his  colleagues  (1998;  Potter,  Roth, 
Woods  and  Elm,  2000)  have  argued,  new  support  technologies 
should  be  regarded  as  hypotheses  about  what  constitutes 
effective  support,  and  how  technological  change  is  expected  to 
shape  cognition  and  collaboration  (Woods  and  Dekker,  2002). 
This  is  a  fundamental  premise  of  the  work-centered  design 
framework. 

A  key  objective  of  a  work-centered  evaluation  is  to 
assess  whether  the  proposed  design  concepts,  as  embodied  in 
the  prototype,  have  the  positive  effects  predicted  by  the 
system  developers  (i.e.,  to  evaluate  the  ‘hypothesized  model  of 
support’).  The  specific  anticipated  benefits  of  the  timeline 
included: 

-  A  decreased  in  time  to  recognize  impacts  of  mission 
changes  during  execution; 

-  A  decreased  number  of  errors  in  replanned  missions  in 
execution. 

The  evaluation  tested  these  hypothesized  benefits  by 
comparing  performance  of  Execution  Cell  personnel  using  the 
timeline  with  performance  using  their  current  Information 
Technology  Systems  on  comparable  scenarios. 

A  second  hallmark  of  a  work-centered  evaluation  is 
that  it  includes  meaningful  measures  of  performance  that  tap 
not  only  the  usability  of  the  system  but  also  the  usefulness  and 
impact  of  the  system  (Eggleston,  Roth  and  Scott,  2003). 
Usability,  usefulness  and  impact  at  different  levels  within  the 
organization  were  assessed  using  a  post-test  questionnaire. 

A  third  hallmark  of  a  work-centered  evaluation 
approach  is  that  it  supports  a  formative  evaluation  function  as 
well  as  a  summative  evaluation.  The  work-centered 
evaluation  aims  to  uncover  additional  demands  and 
unanticipated  requirements  at  the  level  of  work  support  so  as 
to  propel  further  work-centered  design  innovation  (Eggleston, 
Roth  and  Scott,  2003).  This  was  accomplished  via 
observation  of  users  interacting  with  the  timeline  as  well  as 
solicitation  of  user  comments  and  suggestions  as  part  of  the 
post-test  questionnaire. 

An  additional  objective  of  the  evaluation  was  to 
demonstrate  that  program  success  criteria  established  by  the 
Integrated  Product  Team  (IPT)  championing  the  project  were 
met.  Key  Performance  Parameter  (KPP)  success  criteria  to  be 
established  by  the  evaluation  were: 

•  User  Acceptance  (via  post-test  questionnaire):  60% 
participant  acceptance  established  as  a  minimum 
requirement,  90%  as  the  objective; 


•  Improved  mission-related  SA:  Level  2  SA 

(comprehension  of  current  situation)  established  as  a 
minimum  requirement,  Level  3  SA  (ability  to  project 
impact  on  future  sorties  and  missions)  established  as  the 
objective. 

METHODS 

Design 

The  study  used  a  within  subjects  design  to  compare 
the  performance  of  Execution  Cell  personnel  using  the 
timeline  prototype  (timeline  condition)  against  their 
performance  using  their  current  IT  system  displays  (legacy 
system  condition)  on  each  of  five  comparable  scenarios/trials. 
The  order  of  the  two  test  conditions  and  scenarios  presented 
were  counterbalanced. 

Participants 

Twelve  experienced  current,  Execution  Cell 
personnel  volunteers  participated  in  the  study.  Participants 
had  a  mean  of  five  years  experience  in  their  position  with  a 
range  of  .5  to  1 1  years.  While  they  had  all  received  previous 
training  on  the  legacy  system  and  were  expected  to  use  it  to 
perform  their  work,  the  extent  of  actual  use  varied  due  to  some 
preferences  for  previous  versions  of  the  legacy  system  that 
were  still  available  for  use. 

Procedure 

Test  sessions  were  conducted  in  a  closed  office 
located  close  to  the  operations  center  for  the  convenience  of 
the  evaluation  participants.  Each  participant  was  tested 
individually.  A  test  session  lasted  approximately  2.5  hours 
and  included: 

-  Introduction,  demographics  questionnaire,  informed 
consent. 

-  Training  on  timeline:  30  minutes  (immediately  prior  to 
presentation  of  timeline  condition). 

-  Two  Test  conditions 

-  Post  test  questionnaire 

A  fixed  database  containing  a  representative  set  of 
missions  fed  both  a  current  IT  system  client,  and  the  timeline 
server  machine.  Test  participants  used  either  the  legacy  system 
or  the  prototype  timeline  (both  of  which  ran  on  a  laptop 
attached  to  a  desktop  monitor)  to  assess  the  effects  of  mission 
changes  on  the  feasibility  of  completing  a  mission  as  planned. 

Test  participants  saw  different,  but  comparable, 
realistic  work  scenarios  in  the  two  test  conditions.  Each  test 
condition  consisted  of  one  practice  trial  followed  by  5  test 
trials.  Test  scenarios  in  the  trials  included  violations  of 
port/airfield  operating  hours;  quiet  hours;  air  refueling 
reservation  times;  crew  duty  day;  crew  return  time;  and  impact 
on  next  mission. 

Test  scenarios  consisted  of  two  parts  or  phases:  an 
‘initial  call’  specifying  a  delay  in  a  mission  leg,  where  the  test 
participant  was  asked  to  assess  repercussions  of  delay  if  any, 
and  a  ‘suggested  solution’  phase,  where  they  were  presented 
with  a  possible  solution  and  asked  to  indicate  whether  the 
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suggested  solution  eliminated  problems  and/or  introduced  new 
problems.  Initial  call  and  suggested  solution  information  was 
presented  on  index  cards. 

We  recorded  the  time  to  assess  impacts  in  both  the 
initial  and  suggested  solution  portions  of  the  trial  (using  a  stop 
watch)  as  well  as  errors.  Any  usability  issues  in  using  the 
timeline  prototype  were  documented. 

After  each  condition,  participants  filled  out  a  NASA 
TLX  workload  rating  form  (Hart,  S.  G.  and  L.  E.  Staveland 
1988),  and  an  SA  self  report  rating  form  for  that  test  condition. 
The  SA  form  included  likert-rating  scale  questions  for  levels 
1,  2  and  3  SA  as  operationally  defined  for  the  mission  plan 
domain.  At  the  end  of  the  two  test  conditions,  test  participants 
completed  a  post-test  questionnaire  that  included  Likert-rating 
scale  questions  on  usability,  usefulness,  and  impact  on  C2 
organization  objectives  of  the  timeline  prototype.  Participants 
were  also  given  the  opportunity  to  write  in  comments  on  the 
usefulness  of  the  timeline  prototype  and  any  suggestions  for 
improvement  in  the  post-test  questionnaire. 

RESULTS 

Statistical  analyses  were  conducted  comparing 
performance  on  the  test  scenarios  with  the  timeline  with 
performance  using  the  legacy  system. 

Task  Response  Time 

Analyses  of  variance  indicated  statistically  significant 
differences  in  response  time  between  the  timeline  and  legacy 
system  conditions  for  both  the  initial  call  (IC)  portion 
(F=66.29,  df  =  1,1 1,  p  <.0001)  and  suggested  solution  (SS) 
portion  (F=  6.60,  df =1,1 1,  p  <.03)  of  the  test  scenarios.  The 
timeline  condition  led  to  significantly  faster  answers  in  both 
cases  (See  Figure  4). 


Initial  Call  (IC)  Suggested  Solution 

(SS) 


Figure  4.  Mean  response  time  to  assess  repercussions  of 
mission  changes  with  the  timeline  vs.  legacy  system. 

Errors  in  Replanning 

Participants  made  fewer  errors  in  the  timeline 
condition  (See  Figure  5).  The  differences  in  error  rates  were 
statistically  significant  on  the  IC  portion  of  the  scenarios  (e.g., 
Chi-square  =  10.568,  with  1  degree  of  freedom  P  =  0.0012). 
While  error  rates  were  also  lower  with  the  timeline  than  with 
legacy  system  for  the  SS  portion,  the  results  only  approached 


significance  (e.g.,  Chi-square  =  3.01  with  1  degree  of  freedom 
P  =  0.0828). 


Figure  5.  Percent  error  in  answers  to  test  scenario  questions 
with  the  timeline  vs.  legacy  system 

Situation  Awareness 

Figure  6  presents  mean  self  ratings  on  various 
elements  of  SA.  Mean  SA  ratings  were  higher  for  the  timeline 
than  for  legacy  system  in  all  cases.  This  included  elements 
that  tapped  Level  3  SA  (i.e.,  impact  on  future  sorties,  impact 
on  future  missions,  and  plan  changes  needed).  Analysis  of 
variance  indicated  that  the  differences  in  SA  between  the 
timeline  and  legacy  system  conditions  were  statistically 
significant  (F=  54.97,  df=  1 ,1 1 ,  p  <  .0001 ). 


Figure  6.  Mean  self-rated  SA  on  the  various  elements  of  SA 
for  the  timeline  and  legacy  system  (Scale:  1  to  8;  8  =  very  high 
SA). 


Workload 

Perceived  workload,  as  measured  by  NASA-TLX, 
was  significantly  lower  in  the  timeline  condition  than  in 
legacy  system  condition  (F=  6.18,  df=  1,1 1,  p  <  .03).  Mean 
workload  ratings  are  presented  in  Figure  7. 

Acceptance  and  Impact  on  Work 

The  Post  test  questionnaire  ratings  were  examined  to 
assess  test  participant  perception  of  the  usability,  usefulness 
and  impact  of  the  timeline.  Mean  ratings  on  all  questions 
were  positive  (>  4  on  a  1  to  8  scale,  with  1  =  extremely 
negative  and  8  =  extremely  positive).  In  particular,  the 
question  on  overall  acceptability  had  a  mean  of  7.2  and  a 
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range  of  6  to  8.  Mean  ratings  were  also  high  for  questions 
relating  to  impact  on  own  work  (7.25);  impact  on  work  of 
immediate  work  group  (7.25);  and  impact  on  overall  mission 
of  the  organization  (7.08). 


Figure  7.  Mean  NASA  TLX  workload  ratings.  (Scale:  0  to  1  ; 
1=  high  workload). 


Participants  also  gave  the  timeline  high  ratings  in 
response  to  questions  relating  to  effectiveness  of  the  timeline 
in  decreasing  time  to  recognize  impact  of  changes  (7.3); 
improving  ability  to  generate  robust  plans  (7.2);  reducing 
possibility  of  replanning  error  (7)  and  improving  resource 
utilization  (6.5). 

Finally,  participants’  written  and  oral  comments  were 
examined  to  identify  opportunities  for  improvements  to  the 
timeline.  A  complete  list  of  suggested  improvements  was 
documented  and  reviewed  by  the  design  team  for  inclusion  in 
the  prototype  deliverables  and/or  future  design  activities. 

DISCUSSION 

The  work-centered  evaluation  established  a  clear 
performance  improvement  with  the  work-centered  timeline 
visualization  as  compared  with  the  tabular  displays  used  in  the 
legacy  system.  Moreover,  the  results  pointed  to  positive 
benefits  of  the  timeline  prototype  with  respect  to  usability, 
usefulness,  and  impact  on  organization  mission  objectives,  as 
measured  by  the  post-test  questionnaire. 

The  prototype  timeline,  not  only  met,  but  exceeded 
the  performance  KPP  objectives  established  by  the  multi¬ 
disciplinary  IPT.  There  are  current  plans  to  incorporate  the 
timeline  into  future  upgrades  of  the  operations  center  IT 
systems. 

The  study  illustrates  the  methods  and  benefits  of 
work -centered  design  and  evaluation.  The  results  point  to  the 
value  of  taking  a  work-centered  analysis  and  design  approach. 
It  highlights  the  ability  of  work-centered  systems  to  more 
effectively  support  the  key  cognitive  work  tasks. 

The  study  contributes  to  the  growing  body  of 
literature  highlighting  the  importance  of  grounding  design  in  a 
detailed  analysis  of  the  context  of  work  (e.g.,  Vicente,  1999). 

It  illustrates  the  power  of  support  systems  that  incorporate 
visualizations  that  are  finely  tuned  to  the  cognitive  and 
collaborative  demands  of  the  work,  revealing  the  affordances 
and  constraints  in  the  work  domain  that  impact  problem¬ 
solving  and  decision-making  (Burns  and  Hajdukiewicz,  2004). 
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